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CEC: Leading the Way 

The Council for Exceptional Children (CEC) is the largest professional 
organization internationally committed to improving educational out- 
comes for individuals with exceptionalities. CEC accomplishes its 
worldwide mission on behalf of educators and others working with 
children with exceptionalities by advocating for appropriate govern- 
ment policies; setting professional standards; providing continuing pro- 
fessional development; and assisting professionals to obtain conditions 
and resources necessary for effective professional practice. 



CEC: The Unifying Force of a Diverse Field 

The Council for Exceptional Children, a private nonprofit membership 
organization, was established in 1922. CEC is an active network of 59 
State/Provincial Federations, 900 Chapters, 17 Specialized Divisions, 
and 275 Subdivisions with reach in over 40 countries. 



The CEC Information Center: 

International Resource for Topics in Special and Gifted Education 
The Counci' for Exceptional Children is a major publisher of special 
education In mature and produces a comprehensive catalog semiannu- 
ally. Journals such as TEACHING Exceptional Children (published quar- 
terly) and Exceptional Children (published 6 times per year) reach over 
100,000 readers and provide a wealth of information on the latest 
teaching stra tegies, research, resources, and special education news. CEC 
Today, the Council's monthly communications vehicle, provides the 
latest on Association issues, events, and advocacy efforts. 

To help those in the field better understand the value of alternative 
assessment practices, CEC is proud to release Performance Assessment and 
Students with Disabilities: Usage in Outcomes- Based Accountability Systems. 




The Council for Exceptional Children 

1920 Association Drive • Reston, Virginia 22091 
703/62U-3660 (Voice/TDD) 
703/264-9494 (Fax) 
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Foreword 



CEC's policy on inclusive schools and community settings invites all 
educators, other professionals, and family members to work together to 
create eariy intervention, educational, and vocational programs and 
experiences that are collegial, inclusive, and responsive to the diversity 
of children, youth, and young adults. Policymakers at the highest levels 
of state /provincial and local government, as well as school administra- 
tion, also must support inclusive principles in the educational reforms 
they espouse. 

One area in which the inclusion of students with disabilities is 
critical is the development and use of new forms of assessment. This is 
especially true when assessment becomes a tool by which local school 
districts, states, and our nation show accountability for the education of 
students. 

As multidimensional instruments that can cross curriculum areas, 
performance assessments have the potential to be powerful instruc- 
tional tools as well as tools for accountability. As this new technology 
is applied in creating new assessment instruments, students with dis- 
abilities must be considered during the design of the assessment, ad- 
ministration, scoring, and reporting of results. 

CEC is proud to contribute this Mini-Library to the literature on 
performance assessment, and in so doing to foster the appropiate inclu- 
sion of students with disabilities in this emerging technology for instruc- 
tion and accountability. 



/// 

6 



ERIC 




Preface 



Performance assessment, authentic assessment, portfolio assessment — these 
are the watchwords of a new movement in educational testing. Its 
advocates say this movement is taking us beyond the era when the 
number 2 pencil was seen as an instrument of divine revelation. Its critics 
say it is just another educational bandwagon carrying a load of untested 
techniques and unrealisiic expectations. 

Despite the criticisms and reservations that are sometimes ex- 
pressed; these new approaches are being implemented in a growing 
number of large-scale assessment programs at federal state, and district 
levels. They are also finding their way into small-scale use at school and 
classroom levels. 

What about students with disabilities? Are the new assessment 
techniques more valid than conventional assessment techniques for 
these students? Are the techniques reliable and technically sound? Will 
ti.ey help or hinder the inclusion of students with disabilities in large- 
scale assessment programs? Can classrc teachers use the techniques 
to assess student learning and possibly enrich the classroom curriculum? 

The following fictional vignettes illustrate some of these issues. 



Vignette 1 

The State of Yorksylvania developed educational standards 
and a statewide svstem of student assessments to monitor 
progress in achieving the standards. The use of standardized 
multiple-choice tests was rejected because these tests were 
thought to trivialize education. It was feared that teachers 
would "teach down" lo the tests rather than "teach up" to the 
standards. So # committees of teachers, parents, and employ- 
ers were formed to translate the standards into "authentic" 
performance assessments. The resulting assessment system 
was called the Yorksylvania Performance Inventory (YPI). 
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Once a year, students from every school in the state were 
y> ^Administered the YPI, which consisted of several assess- 

0(f) j] \n ^ ments / eac h of which required up to 3 days to complete. 
^ 4-43 Students worked, sometimes individually and sometimes in 
small groups, on tests involving complex, high-level tasks 
that crossed curriculum areas. In one task, students individu- 
ally did research and answered essay questions interrelating 
the geography, wildlife, and history of their state. In another 
task, students worked in groups to design a car powered by 
fermentation. Schools were provided with practice activities 
and curriculum guides to encourage the infusion of perform- 
ance assessment activities into the srhool curriculum. 

The state policy allowed special education students to be 
included in the YPI, excluded, or provided with special modi- 
fications, depending on their individual needs as indicated in 
their individualized education programs. InitiaJJy, most spe- 
cial education teachers supported the YPI because they felt it 
eliminated some r r tificial barriers (reading, test-taking skills, 
etc.) that put their students at a disadvantage on other types 
of tests. However, there were some questions and issues, such 
as the following: 

• Some of the YPI tasks involved a lot of reading, more than 
was found on previous types of tests. 

• Special education teachers sometimes felt pressured to 
exclude their students from testing in order to increase the 
school's scores. 

• Special education students sometimes experienced ex- 
treme frustration in the YPI assessments, many of which 
bore no resemblance to these students' other schoolwork. 

• Some parents of special education students questioned 
whether the standards were really applicable to their chil- 
dren and whether the YPI was diverting instruction from 
more rele\ 't and important topics. 

Vignette 2 

A teacher named Pat had students at a wide range of func- 
tioning levels, including a number of mainstreamed students 
receiving special education services. Pat was always on the 
lookout for new ideas and approaches. Pat began reading 
articles and attending conferences on new assessment ap- 
proaches termed portfolio assessment, authentic assessment, per- 
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formance assessment, and alternative assessment. These ap- 
seemed to make a lot of sense, and Pat decided to 
0 <443 - ^ them out. One of the first approaches Pat tried was 
authentic assessment. Rather than simply testing students on 
their rote learning of skills and content, Pat began to look for 
ways to use realistic, complex activities to test whether the 
students could actually apply what they learned. For exam- 
ple, Pat combined writing, spelling, science, and career skills 
into an activity in which students wrote letters of application 
for jobs as physicists, biologists, or chemists. Pat particularly 
valued activities that engaged students in solving interesting 
problems. For example, after a unit on optics, Pat assigned 
students to draw a diagram explaining why mirrors reverse 
an image from left to right but not from top to bottom. The 
students grappled with that problem for several days. 

Pat liked the holistic scoring procedures developed in 
tnese new assessment approaches. Rather than simply mark- 
ing a response correct or incorrect, Pat scored student work 
on a number of dimensions (e.g., analysis of the problem, 
clarity of communication) according to meaningful quality 
criteria. The development of authentic performance tasks and 
scoring procedures helped Pat clarify the most important 
learning outcomes. 

Pat also liked the idea of portfolio assessment, in which 
students could select and collect "best pieces" to demonstrate 
their learning and achievement during the year. Student 
self-evaluation became a valued part of this process. 

In all, Pat was very pleased with these new assessment 
approaches and intended to continue using them. Instruction 
became more activity based and more focused on real-world 
uses of the material There were, however, some issues that 
Pat began to think about: 

• Students with deficits in certain academic areas, notably 
writing, were at a real disadvantage. It was sometimes 
hard to determine whether an inadequate response re- 
sulted from poor writing skills, poor mastery of the con- 
tent, poor problem-solving skills, lack of creativity, or 
some comhi lation of these factors. Pat considered allow- 
ing some st-idents to tape record their responses, but de- 
cided not to. Wasn't writing itself an authentic task 
required in the real world? 
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Pat wasn't sure how to use the information provided by 
^these tests to plan additional instruction, particularly if a 
student was having difficulty. 

Pat wondered how to tell whether or not an activity was 
in fact authentic, especially for students whose adult lives 
would be very different from Pat's own. 



In 1992, the Division of Innovation and Development (DID) in the 
U.S. Department of Lducation's Office of Special Education Programs 
and the ERIC/OSEP Special Project of The Council for Exceptional 
Children formed a Performance Assessment Working Croup to discuss 
issues such as these. The term performance assessment was adopted as a 
general designation for the range of approaches that include perform- 
ance assessment, authentic assessment, alternative assessment, and port- 
folio assessment. 

Performance assessment was defined has having the following 
characteristics: 

1 . The student is required tocreatcan ansxeerora product rather than simply 
fill in a blank, select a correct an&ivr from a list, or decide whether a 
statement is true or false. 

2. The tasks are intended to be "authentic/' The conventional approach 
to test development involves selecting i. :ms that represent curricu- 
lar areas or theoretical constructs, and that have desired technical 
characteristics (e.g. they correlated with other similar items, they 
discriminated between groups, etc.)- Authentic tasks, on the other 
hand, are selected because they are "valued in their own right" 1 
rather than being "proxies or estimators of actual learning goals." 2 

The Performance Assessment Working Croup produced this seri2s 
of four Mini-Library books on various topics relaU : .o performance 
assessment and students with disabilities. In National and State Perspec- 
tives on Performance Assessment and Students with Disabilities, Martha 
Thurlow discusses trends in the use of performance assessment in large- 
scale testing programs. In Performance Assessment and Students with D/s- 
abilities: Usage in Outcomes- Based Accountability Systems, Margaret 
McLaughlin and Sandra Hopfengardner Warren describe the experi- 



! R. L. Linn, E. L. Baker. & S. B, Dunbar. (1991). Complex, performance-bused assessment: 
Expectations and validation criteria. Educational Researcher, 20(8), 15-21. 
*M. W Kirst. (1991). Interview on assessment issues with Lorrie Shepard. Educational 
Researcher, 20(2), 21 -23, 27. 
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ences of &ate and local school districts in implementing performance 
assessm#wf In Creating Meaningful Performance Assessments: Fundamental 
gj^S^ppfe, Stephen Elliott discusses some of the key technical issues 
involved in the use of performance assessment. And, in Connecting 
Performance Assessment to Instruction, Lynn Fuchs discusses the class- 
room use of performance assessment by teachers. 



Martha J. Coutinho 
University of Central Florida 

David B. Malouf 
U.S. Office of Special Education Programs 

August, 1994 
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and learning disabilities. 
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view of specialeducation, believing that as a practical matter, special and 
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individualized education to meet the special needs of students. To 
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Washington and then to the University of Maryland. 
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partnership between Baltimore City Public Schools and the University 
of Ktoyland. Additionally, she is directing a new effort, under suLcon- 
tr^reto the Maryland State Department of Education, that will develop 
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about changes in practice. She believes that there is a need for systemic 
organizational reform to provide more effective educational experiences 
for youngsters. Her dedication to this belief developed when, as an 
undergraduate, she volunteered at an institution for individuals with 
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To Dr. Warren, "it became clear that the path to a better life for these 
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Her graduate work was in public administration with an emphasis 
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tion for these students," she says. "With the advent of systemic reform 
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||W developing performance assessments for the State of 
faryTand so that students with severe cognitive disabilities are included 
in the accountability system, 

Nfe^^ret McLaughlin and Sandra Warren recently authored issues 
^mhf^hns in Restructuring Schools and Special Education Programs, an- 



other CEC publication. 
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1 . Introduction 



What skills do we expect young Americans to have when they complete 
their education? How do we assess thos« skills, and how will schools be 
held accountable for student attainment of those skills? These are among 
the most commonly asked questions in today's educational reform 
movement. While the answers vary from state to state, a common theme 
is the increasing use of performance assessments and other authentic 
assessments to gauge student progress toward the desired student out- 
comes. 

As part of the educational reform movement, U.S. educators and 
policymakers have endorsed the concept of accountability for student 
outcomes (Brauen, O'Reilly, & Moore, 1994; Kirst, 1990). States and local 
districts have implemented outcomes-based accountability, which em- 
phasizes the central role of student outcomes as a means of assessing 
school performance. The accountability systems typically incorporate 
multiple indicators and multiple forms of assessment; however, the core 
of these systems revolves around student performance. Outcome meas- 
ures include students' test scores, self-evaluations, and teacher or peer 
evaluations. 

What Is Accountability? 

Hill and Bonan (1991) and Kirst (1990) have defined accountability as a 
systematic method to ensure that those working in a school system are 
moving in desired directions and thus are responsible to their constitu- 
ents. Outcomes-based accountability means that the educational system 
is responsible for student attainment of specific learner outcomes. It 
differs from more traditional forms of accountability that have empha- 
sized compliance with procedures or practices or have focused solely on 
input measures such as the resources allocated or expended for educa- 
tion (Brauen et aL, 1994). Outcomes-based accountability requires more 
than assessing student educational outcomes or collecting other student 
data such as school participation and school completion rates. For 
schools to be held publicly responsible, student achievement, attendance 
data, and similar indicators of school performance must be reported and 
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against certain standards or criteria. In addition, account- 
HSly involves some explicit or implicit consequences for 
schools (e.g., principals or school staff) or individual students. For 
school^these consequences can be in the form of rewards or sanctions 
thaW^administered based on how well students are meeting certain 
l^fermance standards. For students, results of assessments are fre- 
quently linked to receipt of a high school diploma. 



The premise of outcomes-based accountability is that 
school staff have to be guided by the cognitive, 
behavioral, and attitudinal results they produce in the 
students they teach. 



In sum, the premise of outcomes-based accountability is . . . 
that school staff have to ... be guided by the cognitive, 
behavioral, and attitudinal results they produce in the stu- 
dents they teach. Outcomes-based accountability also means 
that outcomes not only must be measured, but they must 
have some consequences. (Brauen et al., 1994, p. 3) 

Issues in Linking Student Outcomes to School 
Accountability 

A number of issues must be addressed if schools are to be held account- 
able for student outcomes. These include questions of (a) how to define 
the valued outcomes, (b) how to assess the outcomes, and (c) what 
standards will be applied. It is also important to determine the conse- 
quences for individuals representing the school systems as well as the 
consequences for students. 

These issues are posing major challenges to states and local school 
districts across the country. For the past two decades, individual states 
have been moving toward developing a systems approach to improving 
educational outcomes for students. By 1992, 46 states had developed 
some type of systemic approach to evaluating student outcomes (Coun- 
cil for Educational Development and Research, 1994). These endeavors 
have resulted in a variety of approaches to defining and assessing 
student outcomes. Some are very specific and relate to academic content 
areas, while others define learning or thinking processes or social or 
emotional behaviors (Warren, 1993). However, two common charac- 
teristics have pervaded virtually every outcome assessment system 
developed by the states: (1) an emphasis on increasing the expectations 
for student performance through the development of standards and 
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hp^jillffpthe consequences attached to that performance (Brauen et 
hi, 1994; Warren, 1993). 
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itabiiity for Outcomes of Students with Disabilities 



Tne current general education activities related to outcomes-based ac- 
countability have major implications for students receiving special edu- 
cation services. Special educators are increasingly concerned with the 
outcomes of students who are identified as having disabilities and 
receiving special education support services (Hasazi, Gordon, & Roe, 
1985; Jakweth & Frey, 1992; National Center on Educational Outcomes 
[NCEO], 1991; National Council on Disability, 1993; Wagner, 1991). 
These studies have examined the postschool status of varied groups of 
students with disabilities. The outcomes reported in these studies in- 
clude higher than desired rates of unemployment, as well as low wages 
and limited community or social independence. These disappointing 
outcomes have prompted special education policymakers to call for 
increased attention to the results of special education programs. 

Historically, special education program accountability has focused 
on determining whether a service has been provided as documented in 
the individualized education program (IEP) and on other procedural 
compliance indicators buch as the timeliness of assessments and parental 
notifications. Whatever educational outcome data have been reported 
are incorporated into the IEP. However, research (Olsen & Massanari, 
1991; Smith, 1990) indicates that the IEP has evolved into little more than 
o compliance document in many school districts. It documents services 
and placements but is of little use for instruction or evaluation. More- 
over, the IEP was never designed to monitor progress toward long-term 
program outcomes. 



The level of rigor applied to the assessment of outcomes 
for nondisabled students must also be applied to students 
who are receiving special education services. 



As noted by DeStef ano ( 1993), efforts to document the ef c ?ctiveness 
of special education programs by assessing student attainment of long- 
term goals are almost totally nonexistent. DeStafano and others 
(McGrew, Thurlow, Shriner, & Spiegel, 1992; McLaughlin & Warren, 
1994) believe that educational systems must be accountable for all stu- 
dents and that the level of rigor applied to the assessment of outcomes 
for nondisabled students must also be applied to students who are 
receiving special education services. Furthermore, there must be equally 
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|p||^||^|f |||Iards for performance, and there must be consequences for 
Ihe educational systems. 

Lacking a system of comprehensive accountability, the educational 
systenpUs no way of knowing whether or not a particular program is 
^if^iVe in educating students with disabilities. However, account- 
ability is dependent upon having a system of assessments that is flexible, 
can accommodate diverse learners, and produces information that is 
valid and reliable Qakweth & Frey, 1992). 
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2. Assessment of Student 
Outcomes 

Increasingly, outcomes-based accountability systems are using results 
of both traditional assessments, such as norm- referenced tests, and 
authentic or performance assessments in an effort tn measure broad 
domains of student knowledge. 

Traditional Assessment 

Norm-referenced tests have long been used to provide educators and 
policymakers with information on student progress. This form of testing 
is useful in ascertaining student skills across a large number of domains, 
in a relatively short period of time, and in a cost-effective manr^ . 
Furthermore, these tests are administered and evaluated under uniform 
conditions so that results can be compared across groups ct students 
(Office of Technology Assessment 1992). 

Although multiple-choice and true/ false questions are typically 
used to describe more traditional assessments, it is important to note that 
a variety of other testing formats may fall into this category. Examples 
could include oral examinations and written essays (Office of Technol- 
ogy Assessment, 1992). 

Use of traditional assessments has been increasingly criticized in 
recent years because they lack integration of instruction and testing; 
emphasize factual knowledge and discrete skills; force the selection of 
one right answer versus exploration of multiple possibilities; require 
artificially short answers; and force students to work in isolation (Linn, 
Baker, & Dunbar, 1991; Office of Technology Assessment, 1992). In 
addition, traditional assessments have been challenged as being unfair 
to individuals of diverse cultures and learning styles (Rotfunan, 1991). 

Moreover, these types of assessments are inconsistent with many 
of the current educational outcomes being defined by states. For exam- 
ple, student outcomes such as making judgments, solving problems, 
reasoning, or communicating for multiple purposes would be difficult 
to measure in traditional multiple-choice or short-answer formats. 
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IHnent of a student's ability to demonstrate reasoning, 
gather information from multiple sources, and contribute to collabora- 
tive efforts requires a multifaceted approach. 

fea^sponse to these concerns, a growing emphasis has been placed 
^||3f e ^°pi n & student assessments that can measure knowledge and 
competencies in ways that require students to demonstrate the higher- 
order process skills as well as content knowledge. 



Issues in Using Performance Assessments in 
Large-Scale Gn-Demand Assessment Programs 

In response to the desire to measure student abilities in more complex 
and integrated learning environments, many slates and local districts 
have turned to alternative assessments that provide a broader measure 
of student knowledge and c kills. Of particular interest is the use of 
performai ice assessments that measure the extent to which students can 
integrate knowledge, apply it to meaningful and complex tasks, and 
produce a response, whether through demonstrations, constructions, or 
performances (Poteet, Choate, & Stewart, 1993). 

Performance assessments may take a variety of forms, including 
essays, hands-on science problems, open-ended problems, computer 
simulations of real- world problems, artistic productions, and portfolios 
of student work. The assessments require the performance of an activity 
that integrates knowledge from a variety of domains (i.e., science, mathe- 
matics, reading, writing) and results in a product — either written, con- 
structed, verbal, or performance (Office of Technology Assessment, 
1992; Shavelson, Baxter, & Pine, 1992). Evaluations are based on the 
"right" answer as well as a defense for the rationale or process for 
developing the response (Shavelson et al„ 1992). 

Well-designed performance assessments can give teachers and 
students instructionally relevant information so the results can be used 
to adjust instruction (Office of TechniJogy Assessment, 1992). Thus, 
policymakers view the use of these tests as a way to directly influence 
curriculum and teacher behavior. The reasoning is that if schools are held 
accountable for students' proficiencies, then teachers will teach those 
critical skills and behaviors. 

A push for a national commitment to performance assessment is 
being spearheaded by FairTest: The National Center for Fair and Open 
Testing (Rothman, 1991). Representatives of over two dozen education, 
civil rights, and advocacy groups were called on by FairTest to work 
toward the development of a national assessment system focusing on 
reliable performance assessments that are sensitive to the cultural biases 
found in existing assessments (Rothman, 1991). The ultimate goal of this 
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Effitalitlptti^Wfcsure that national assessments directly address and 



sensitive to cultural biases. 




|gp@ri used for accountability purposes, the assessments 
must be conducted with a large number of students, there 
must be consistency in the domains of knowledge being 
assessed, and the assessments must yield adequate 
samples of student performance within those domains. 



However, there are a number of challenges when performance 
assessments are used as part of large-scale federal, state, or district-wide 
assessment programs. When used for accountability purposes, the as- 
sessments must be conducted with a large number of students, there 
must be consistency in the domains of knowledge being assessed, and 
the assessments must yield adequate samples of student performance 
within those domains. Costs and time associated with administration of 
the assessments to large numbers of students are also major considera- 
tions. Finally, reliability in scoring the assessments becomes critical, 
especially if the results are linked to sanctions or rewards for schools or 
consequences for students, such as the award of a diploma. Two issues 
that are particularly troublesome when considering large-scale use of 
performance assessments are costs and scoring. 



Performance assessments and other forms of authentic assessments have 
one significant drawback when used as part of large-scale on-demand 
assessment programs: their cost. Authentic assessments are costly to 
develop as well as to administer and score. While some of the expense 
lies with equipment and supplies to be used in the performance tasks, 
the majority of the costs are incurred in personnel. Significant invest- 
ments of human capital are required, particularly in the areas of task 
administration and scoring of the tasks (Office of Technology Assess- 
ment, 1992; Popham, 1993). 

One approach to maximizing the balance among time, cost, and 
desire to obtain wide-scale data on the progress of students throughout 
a program or system is the concept of matrix sampling (Popham, 1993). 
This approach enables assessment and accountability designers to sys- 
tematically sample an array of students and performance tasks. No 
student performs all tasks, yet there are sufficient numbers of perform- 
ance tasks over all to enable the system to evaluate programs. Such 
sampling reduces costs and time burdens but limits the ability to link 



Costs 
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!D«lA students, since not all students are assessed. This 

limits the use of the assessments for improving instruction of individual 
^ students. 

^coring the assessments is also costly. Because student perform- 
is not reduced to a correct answer on a test form, a number of 
individuals must be trained and employed as assessors. This, m turn, 
requires investment of resources to train the assessors. 



Reliability in scoring performance assessments is a major 
issue, particularly when these forms of assessment are 
used in large-scale programs. 



Scoring 

Many of the issues surrounding the scoring of performance assessments 
have been discussed in other books in this Mini-Library, Reliability and 
validity are two critical aspects of effective assessments. Reliability in 
scoring performance assessments is a major issue, particularly when 
these forms of assessment are used in large-scale programs. Because 
teachers oc other trained assessors are asked \o rate diverse types of 
student work, intensive training is required. Howev er, some (Baker & 
Linn, 1993; Dietel, 1993) believe that current scoring techniques may not 
be strong enough to warrant use of some performance assessments for 
school-wide accountability. As evaluators search for innovative ways to 
assess students' abilities to integrate knowledge and skills in meaningful 
ways and to demonstrate higher-order thinking skills, it becomes more 
challenging to ensure that assessments are reliable and valid. 
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3. Issues in Using Performance 
Assessments for Students 
with Disabilities 



Performance assessments can be extremely relevant in 
the students' lives while offering educators the 
opportunity to evaluate student progress and program 
effectiveness. 



Performance assessments offer a number of benefits over the use of 
traditional standardized assessments for students with disabilities 
(Coutinho & Malouf, 1993; Poteet, Choate, & Stewart, 1993). Perhaps 
most important is the potential for linking instruction and assessment. 
Since authentic performance assessments are based on performance of 
valued skills, the experience of completing a task may be intertwined 
with the instructional process. Thus, performance assessments can be 
extremely relevant in the students' lives while offering educators the 
opportunity to evaluate student progress and program effectiveness. 

However, use of performance assessments as part of a system's 
accountability plan raises a number of issues for students with disabili- 
ties. These include (a) defining the outcomes to be assessed, (b) develop- 
ing performance standards, (c) developing assessment accommoda- 
tions, and (d) scoring. 

Defining the Outcomes 

Assessment programs are constructed to measure progress toward val- 
ued educational outcomes, When outcome frameworks are defined too 
narrowly (e.g., academic content domains) and neglect other valued 
areas (e.g., vocational skills, personal management, social skills, and 
communication), the outcomes may not reflect all of the skills and 
competencies that are considered appropriate for students with disabili- 
ties. In particular, students with significant cognitive disabilities may 
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areas as the .urriculum for these students has focused on developing 
functional life skills. As a result, most students with moderate to signifi- 
car^ognitive disabilities have not had access to the instruction that 
=4^cpares them for successful completion of the assessment tasks. More- 
over, the tasks are not relevant to the students' program goals. 



Developing Performance Standards 

Standards are the benchmarks against which student performance may 
be compared. A critical decision in designing an assessment system is 
whether students will be compared to themselves to determine change 
in their performance over time, or whether they will be compared to 
fixed standards of performance (Brauen et al., 1994). 



A critical decision in designing an assessment system is 
whether students will be compared to themselves to 
determine change in their performance over time, or 
whether they will be compared to fixed standards of 
performance. 



Fixed standards promote high expectations for all students and set 
criteria that can be used consistently across scoring rubrics. Many stu- 
dents with disabilities cannot meet some of the absolute standards, 
particularly inacademic skill areas. Where such standards exist, students 
with disabilities may be excluded or discouraged from participating in 
the accountability system. When participation in the assessment pro- 
gram is linked to high school diplomas, students with disabilities may 
be at a particular disadvantage. 

Assessment Accommodations 

Many students with disabilities will require accommodations to partici- 
pate in large-scale assessment programs. These accommodations may 
include additional time to complete the task, alternative testing loca- 
tions, alternative means of administration (e.g., reading, interpretation, 
braille), and alternative supplies or equipment (e.g., computers). Some 
students, particularly those with significant cognitive disabilities, may 
require alternative forms of assessments. While the provision of these 
accommodations will result in a more inclusive assessment system for 
all students, there is concern that the accommodations will be costly and, 
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more important, that they will compromise the integrity of the assess- 
it Ipl WW rdize the reliabmt y of the results. 

Scorino^ 

RjdatTOTo assessment accommodation is the issue of scoring. When 
4rra&«sment results must be reported in the aggregate and when results 
matter, scoring reliability becomes more critical. Assessment experts are 
reluctant to permit accommodations because they believe they compro- 
mise the assessments. Rigorous scoring is as necessary in alternative 
assessments as in traditional assessments. However, the nature of stu- 
dent products and performances for students with moderate to signifi- 
cant disabilities who participate in these alternative programs makes it 
extremely difficult to develop scoring rubrics that allow assessments to 
be scored reliably. 
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4, Performance Assessments 
in Action 



Performance assessments are being incorporated into accountability 
systems with increasing frequency. A number of state and local school 
districts have adopted this approach as either the sole method of assess- 
ment or in conjunction with traditional norm-referenced tests. Many 
states and local school districts have included students with disabilities 
in their accountability systems (see ThurL «w, 1994). In an effort to under- 
stand the implications of including students with disabilities in these 
assessment programs, telephone interviews were conducted with repre- 
sentatives of three states (Kentucky, Maryland, and Vermont) and two 
local school districts (Littleton, Colorado, and Arlington Heights, Illi- 
nois). These jurisdictions were selected because of the extent of their 
commitment to using performance assessments for statewide or district 
accountability. 

Interviews were conducted with key informants in each of the 
states or local districts to obtain information relative to the general use 
of performance assessments, as well as specific issues related to students 
with disabilities. In addition, documents describing the systems were 
reviewed. Following are brief descriptions of the five assessment pro- 
grams. 

Kentucky 

Description of Accountability System 

The Kentucky Educational Reform Act (KERA) outlines six performance 
goals that all students are expected to attain upon graduation from 
Kentucky schools (see Table 1). Individual schools have been granted 
significant latitude in determining how students will master the 75 
state-developed student outcomes. These outcomes apply to all stu- 
dents, regardless of the curriculum they may be receiving (special edu- 
cation, vocational, or academic), and they include academic as well as 
personal management and social skills. 
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TABLE 1 

Characteristics of Selected Accountability Systems Using Performance Assessments 



State 


Outcomes 


Assessments 


How Results 
Are Used 


Inclusion of Students 
with Disabilities 


Kentucky 


Six performance goals include 
communication and math, core 
concepts from the sciences, arts, 
humanities, social studies, and 
practical Living studies; 
self-sufficiency; membership in 
family, work group, or 
community; thinking and 
problem solving; and connecting 
and integrating knowledge. 

Seventy- five outcomes in the 
areas of reading, writing, 
mathematics, science, social 
studies, arts and humanities, and 
vocational education /practical 
living. 


Portfolios in writing and math for all students in 
grades 4, 8. and 12; 

Performance events for all students in grades 4, 8, 
and 12. Focus on mathematics, science, social 
studies, arts and humanities, and vocational 
education/practical living; 

Transitional assessments (open-ended and 
multiple-choice questions) for all students m grades 
4, 8, and 12. Focus in reading, writing, mathematics, 
science, social studies, arts and humanities, and 
vocational education /practical living; or 

Alternative Portfolios for students with severe 
disabilities in grades 4, 8, and 12. Maximum of 2% of 
the school enrollment may develop Alternative 
Portfolios. 


School report 
cards. 

School rewards 
and sanctions. 


Ail students are required 
to participate in the 
transitional assessments 
or Alternative Portfolios 
unless a physician 
provides a statement 
documenting significant 
negative impact on the 
student's health as a 
result of participating. 


continues 
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State 


Outcomes 


Assessments 


How Results 
Are Used 


Inclusion of Students 
with Disabilities 




Content outcomes of reading and 
writing, mathematics, social 
studies, and science. 

Overarching outcomes of 
individual and group problem 
solving, decision making, and 
issue analysis. 


Maryland State Performance Assessment Program 
(MSPAP) (matrix sampling selection of students in 
grades 3, 5, 8). Areas include reading, mathematics, 
writing and language usage, social studies, and science; 

Comprehensive Test of Basic Skills (CTBS) (matrix 
sampling of students in grades 3, 5, 8); 

Maryland Functional Tests (to be passed by graduation). 
Areas include reading, mathematics, writing, and 
citizenship; or 

Proposed: Alternative Performance Assessments for 
students with significant cognitive disabilities ages 9, 11, 
14, 17, and 21. 


Annual school 
performance 
reports. 
Annual school 
district perfor- 
mance reports. 
Annual state 
performance 
reports. 
Student 
diplomas. 
School rewards 
and sanctions. 


Students with 
disabilities may be 
exempted by their 1EP 
team from participation 
in MSPAP, CTBS, and 
Functional Tests. 
Students with 
significant cognitive 
disabilities are eligible 
to participate in the 
Alternative 
Performance 
Assessments. 


Vermont 


Fields of knowledge outcomes 
include mathematics; life science; 
physical science; earth science; 
technology education; history and 
philosophy of science; and career 
preparation in sciences, math, and 
technology. 

Vital results outcomes include 
communication, reasoning and 
problem solving, personal 
development, and social 
responsibility. 


Uniform Assessment (math) includes two 40-item 
multiple-choice tests and a single on-demand task for 
students in grades 4 and 8; 

Uniform Assessment (writing) requires students (grades 
4 and 8) lo respond to a single prompt in a 90-minute 
period; and 

Math and writing portfolios for students in grades 4 and 
8. 


Supervisory 
union reports. 

State reports. 


School-based staffing 
teams determine 
whether or not students 
with disabilities are 
eligible for exemption 
from the portfolio 
development and 
assessments. Exempted 
students may build 
portfolios that do not 
conform to the state 
requirements for use in 
instruction. 


cum (mires 
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State 


{£ 

Outcomes 


Assesstuents 


How Results 
Are Used 


Inclusion of Students 
with Disabilities 


Littleton^ 

cow 


Seven system outcomes that 
include the ability to gather, 
pmress, and communicate 
information through reading, 
written and oral expression, and 
active listening; mathematical 
concepts and processes; cultural, 
scientific, and technological liter- 
acy; critical thinking, problem- 
solving and decision- making 
skills, and intellectual curiosity; 
creative expression; physical and 
emotional well- being; and 
self-esteem and respect for others. 


Individual schools or school clusters are permitted to 
develop their own performance assessments, which 
must link to the seven system outcomes. 


School report 
cards. 

Student 
diplomas. 


Students with 
disabilities who spend 
50% or more of the day 
in the regular class- 
room are expected to 
participate in the 
assessment program. 


Arlington 
Heights, I L 


Eleven General Learner 
Outcomes (GLOs) include 
communication, social 
interaction, analysis, problem 
solving, value judgments and 
decisions, creativity, civic 
responsibility, global 
environment wellness, 
tuchnology, and life and career 
planning, 


Standardized achievement tests (Iowa Test of Basic Skills 

in grades 2, 5, and 8; and Tests of Achievement and 

Proficiency in grades 9 and 11); 

Standardized ability measures (Cognitive Abilities Test 

in grades 2, 5, 8, 9, and 11); 

Integrated Reading and Writing performance 

assessments (grades 4 and 8); 

Writing assessment (grade 12); and 

A variety of school-created performance assessments 

administered in grades K-12. 


Individual 

student 

matrices. 

Student 
diplomas. 


Students with disabil- 
ities may be exempted 
from the traditional 
assessments. However, 
since the performance 
assessments are school 
created, all students are 
expected to participate 
in assessments that 
have been developed 
based on their individ- 
ual skills and needs. 



•Program dismantled during 1W3-1994 school year due to political pressure. 

3i 
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hool report cards indicate progress toward 
meeting the state outcomes by listing the percentage of 
students scoring at each of four performance levels 
gribwce, apprentice, proficient, distinguished) in each of 
four subject areas (reading, math, science, antisocial 
studies). 



School districts are given the flexibility to develop instructional 
content and approaches appropriate {or their students. The Kentucky 
Instructional Results Information System (K1RIS) develops an array of 
assessments to determine student progress. Currently, progress is evalu- 
ated based on performances in the KIRIS assessments or the Alternate 
Portfolio Project, a multidisciplinary approach to evaluating student 
progress in meeting the valued outcomes. 

The local school is held accountable. Annual school report cards 
indicate progress toward meeting the state outcomes by listing the 
percentage of students scoring at each of four performance levels (nov- 
ice, apprentice, proficient, distinguished) in each of four subject areas 
(reading, math, science, and social studies). 

Assessments 

KIRIS contains three types of assessments; (1) portfolios and exhibitions, 
(2) performance-based events in which students apply what they have 
learned to a real situation, and (3) transitional assessments including 
multiple-choice and open-ended questions. Participation in the state-de- 
veloped mandated components of KIRIS are administered in grades 4, 
8, and 12. The state is also developing assessments for use on a voluntary 
basis in all grades. 

For the 1991-1992 school year, Kentucky students participated in 
the writing portfolio process; transitional assessments (i.e., multiple- 
choice, open-response testing in mathematics, social studies, science, and 
reading); and performance events testing mathematics, science, and 
social studies. During the 1992-1993 year, students also developed 
mathematics portfolios. Plans for subsequent school years were to focus 
on the development of integrated, holistic portfolios. 

The vast majority of students with disabilities participate in this 
mainstream assessment system. However, a small percentage (l%-2%) 
of students with disabilities are selected to participate in the Alternate 
Portfolio Project. These students generally have moderate to significant 
cognitive disabilities and are not working toward a high school diploma. 
Students are selected to participate entirely in one system or the other. 
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1992-1993 school year, students with severe disabilities 
v^paftfapafcd in the Alternate Portfolio Project. Their entries include 
a written or dictated letter of introduction describing the portfolio; 
relevan^cadcmic work; schedules reflecting time with nondisabled 
^^§r presume of job experiences; samples of their present mode(s) of 
Communication; and letters of validation from educators, family mem- 
bers, and other supporters. Twenty-eight outcomes were selected as 
preliminary targeted outcomes for the students participating in the 
Alternate Portfolio Project. Additional outcomes will be added each year 
until all students' portfolios include work reflecting progress toward all 
75 valued outCL.. les. 



Exemptions 

All students, including those receiving instruction in a home or hospital 
setting, are required to participate in the KIRIS or Alternate Portfolio 
Project unless a physician provides a statement documenting significant 
negative impact on the student's health as a result of participating. These 
students are exempted from participation in the assessment and ac- 
countability aspects of KERA. 

Accommodations 

A wide array of accommodations is permitted for students with disabili- 
ties. The state education agency (SEA) has provided significant latitude 
in identifying acceptable accommodations, the critical issue being that 
the IEP team must approve the accommodations and they must be used 
by the student on a daily basis in the course of ongoing instructional 
activities. 

The primary focus during portfolio evaluations is the student's 
ability to perform a series of tasks that result in a routine that is mean- 
ingful in the community. Therefore, less focus is placed on supports used 
by a student (e.g., calculators, dictionaries, electronic spe 1 eckers) 
than would be found in more traditional assessments. Accommodations 
provided by nondisabled peers (e.g., student peers, coworkers, store 
workers) are strongly encouraged. 



General Comments 

Educators and advocates in Kentucky have learned that the new ap- 
proaches to assessing student performance have created a need for new 
ways of teaching. This has resulted in a need for teacher training. 
Initially, training focused on discrete topics related to the curriculum in 
1-hour workshops, but teachers are now beginning to view these ses- 
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IE feint IpfcSi lify as vehicles to acquire new knowledge but also as oppor- 



tunities to reflect on instruction and share ideas with colleagues. 

Experiences with KIRIS activities have promoted some major 



to become more community based in their instructional practices. While 
community -based instruction and inclusive education started in re- 
sponse to the assessments, many educators are seeing the inherent value 
of these for students with disabilities and those with diverse learning 
styles. 

Maryland 



A School Performance Review System serves as the 
accreditation body to reward schools for outstanding 
performance, recognize those that are progressing, and 
implement sanctions for those that are not meeting 
standards. 



Description of Accountability System 

The Maryland School Performance Program (MSPP) was developed as 
a comprehensive student outcomes accountability system. Reflecting 
state-level goals and strategies (see Table 1), student learning outcomes 
have been developed in the areas of reading and writing, mathematics, 
social studies, and science. 

MSPP communicates results to the public through school report 
cards (Annual School Performance Reports) that provide data related to 
student population characteristics, student participation rates, and the 
results of selected student assessments. School reports are aggregated 
for each local school district. Outcomes data for students with disabilities 
participating in MSPP assessments are aggregated with those of their 
nondisabled peers. 

A School Performance Review System serves as the accreditation 
body to reward schools for outstanding performance, recognize those 
that are progressing, and implement sanctions for those that are not 
meeting standards. Progress benchmarks have been established for in- 
dividual schools with the expectation that ail schools will reach state 
standards by 1995, and sanctions— including takeovers by local school 
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institution) — will be imposed on schools that do not 
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ra&^ssment of student outcomes within the MSPP includes both norm- 
referenced tests (i.e., Comprehensive Test of Basic Skills [CTBS1), crite- 
rion-referenced tests (i.e., Maryland School Performance Assessment 
Program [MSP API), and the Maryland Functional Tests. 

The CTBS is administered to a matrix sampling of 3rd, 5th, and 8th 
grade students.The Maryland Functional Testing Program is a series of 
four criterion-referenced tests designed to ensure that students possess 
basic competencies and functional skills in mathematics, reading, writ- 
ing, and citizenship. Students must pass each of the four tests as a 
requirement forhigh school graduation. With the exception of the Mary- 
land Writing Test, multiple-choice items are the primary response mode. 

The MSPAP is a collection of performance assessments created to 
provide a mechanism for measuring the desired student outcomes iden- 
tified within MSPP. Performance tasks are administered to a matrix 
sampling of students in grades 3, 5, and 8, unless they are exempted from 
participation (i.e., due to disability or limited English proficiency). The 
tasks are designed to measure what students have learned based on the 
identified state outcomes. The assessments are integrated across subject 
matter content and emphasize the thoughtful application of knowledge 
and skills in content areas. For example, a task for 3rd graders assesses 
student skills in social studies and language usage through the analysis 
of a scenario of welcoming a new student from Japan. Students read 
excerpts on Japanese schools, and are asked to construct maps of the new 
school, develop a schedule that includes comparisons to the Japanese 
schedule, and write an analysis of the adjustments the student will face 
in the new school. Each assessment requires multiple student responses 
that can include open-ended responses, writing mathematical findings, 
and group activities. Results are not reported for individual students. 
Rather, they are used to provide descriptive data about a school's 
performance in the assessed areas. 

Exemptions 

Although the outcomes developed by the state are sufficiently broad to 
include most students, MSPAP assessments reflect a focus on higher-or- 
der, critical thinking skills within specific academic areas. As a result, 
they do not match the curriculum goals of some students. Therefore, 
exemptions may be granted for non-diploma-seeking students with 
disabilities and (on a one-time basis) students with limited English 




)|jf MfS^. Exemptions for students with disabilities are made on a 
student-by-student basis through the IEP process. 

An Alternative Performance Assessment system is being devel- 
opMfey the state for students with significant cognitive disabilities who 

j0 aje^exempted from components of MSPP. The core of these assessments 
will be portfolios, but they will include on-demand performance assess- 
ments that will evaluate student performance and program supports in 
four domains: vocational, recreation /leisure, personal management, 
and community. It is anticipated that these assessments will become a 
part of MSPP and the Annual School Performance Reports for each local 
district, thereby ensuring accountability of special education programs 
and services for students with significant cognitive disabilities. 



Accommodations 

The Maryland State Department of Education, in collaboration with a 
network of special education teachers, has developed a list of permissible 
accommodations for each of the MSPP assessments. Accommodations 
(e.g., scheduling, setting, equipment, presentation, and response) are 
permitted only when they do not invalidate the assessment for which 
they are granted. Accommodations are to be based on individual student 
needs rather than category of disability, level of instruction, or other 
group characteristics. The accommodations are developed during the 
annual IEP team meeting and are to be used throughout the course of 
ongoing instruction during the school year. 



Placing accountability for student achievement directly at 
the school level is creating a new sense of ownership for 
students and a greater awareness among school staff of 
the unique skills and needs of many students and how 
these affect school performance results. 



General Comments 

While the Annual School Performance Reports are key to school restruc- 
turing, the MSPAP is viewed as the vehicle for effecting change in 
instructional strategies to improve student outcomes. Teachers are talk- 
ing of new educational values (e.g., critical thinking skills and integrated 
thinking) and new ways of teaching and testing students (e.g., perform- 
ance-based assessment and development of meaningful products). 

MSPP is having a significant impact on the process of school-site 
restructuring throughout the entire state. Placing accountability for stu- 
dent achievement directly at the school level is creating a new sense of 
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idents and a greater awareness among school staff of 
the&Tque^sTaTfs and needs of many students and how these affect school 
performance results. 

Description of Accountability System 

The Vermont Assessment Program was implemented statewide during 
the 1991-1992 school year. Vermont was the first state to adopt portfolios 
as part of a statewide assessment program. The program employs both 
standardized assessments and portfolios to collect information about the 
performance of 4th and 8th graders in mathematics and writing (see 
Table 1 ). Every student is given an opportunity to build a portfolio. The 
portfolio scoring establishes clear performance standards, and portfolios 
are scored by trained teachers using standard rubrics. Portfolios are 
evaluated according to several criteria on a four-point scale. They in- 
clude classroom work completed throughout the year, as well as a "best 
piece" chosen by the student. 



Assessments 

Information about the statewide performance of students in mathemat- 
ics is gathered from the Uniform Assessment and portfolios. The Uni- 
form Assessment includes two 40-item multiple-choice tests and a single 
on-demand task that the student must complete independently. Each 
student's portfolio is expected to include 10 to 20 items. From these, 
students select 5 to 7 "best pieces" to be scored. The mathematics scoring 
system evaluates student responses using four criteria related to prob- 
lem solving and three related to mathematical communication. For each 
criterion, each of the four performance levels is identified with a bench- 
mark or student paper that serves as an example of a performance rating 
of 1/2, 3, or 4. 

The writing assessment contains information from three sources: 

1. The Uniform Assessment requires all students in a grade level to 
respond to a single writing prompt within 90 minutes. 

2. The "best piece" is selected by the student from the portfolio 
contents to represent his or her best work as a writer. 

3. The remaining contents of the portfolio are scored together and 
referred to as the "portfolio." 

All student writing is evaluated on the basis of criteria on five 
dimensions: purpose, organization, detail, voice/ tone, and gram- 
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I ftilit IpUitese/mechanics. The writing assessment describes levels of per- 



formance in terms of the frequency with which students display the 
skUls; rarely, sometimes, frequently, or extensively. For each criterion 



ision, student papers have been selected to serve as benchmarks or 



Scores for portfolios are reported at the supervisory district level 
In the future, scores for the uniform writing piece will be reported school 
by school. However, the results of the portfolio assessments are clearly 
tied to instruction. Scoring standards define for teachers, students, and 
parents what is expected in the two subject areas, and the standards are 
being used to modify individual students' programs. 

Exemptions 

The SEA wants all students to participate in the portfolio assessment, but 
the basic staffing team in each school can decide whether participation 
would be harmful or the portfolio content is not relevant for a particular 
student. Reportedly very few students are exempted, but each district 
handles this differently. Students with disabilities who are exempted 
may build portfolios that do not conform to the state requirements. These 
are scored by the student's teacher and are used only tor instruction. 
Local school personnel report that this process occurs infrequently. 

Accommodations 

Accommodations are permitted for any student as long as the specific 
accommodation is used as part of instruction. For example, if a student 
uses a computer for writing instruction, then the student may use the 
computer for any part of the writing tasks. 



Teachers reported that portfolios had both positive and 
negative effects on the performance of students of low 
ability. 



General Comments 

There is reportedly little resistance among teachers and administrators 
to the use of the portfolio assessments, because many schools had been 
using them for some time for instructional improvement in grades K 
through 12. The assessments are perceived as ways of improving indi- 
vidual student educational programs, not as ways of evaluating schools. 
There is also little reported resistance to the notion of accountability. 





fx illustrations of each performance level. 
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Vermont's assessment program is being thoroughly evaluated 
imdeij a* contract with the RAND Corporation (Koretz, Stecher, & 
the 1991-1992 pilot year, evaluators administered 
questionnaires to participating teachers and mathematics portfolio rat- 
ers. Additional teacher questionnaires were administered and inter- 
views misconducted with principals during the first year of statowide 
' i^|§nr^ntation (1991-1992). Interim analyses of these data indicated 
that portfolio raters had mixed reactions to the portfolio scoring process. 
Portfolio raters for both 4th grade and 8th grade were concerned about 
the unevenness of tasks that were submitted; they either were difficult 
to fit into a scoring category or were insufficient. Portfolio raters for 4th 
grade also had a difficult time actually scoring the tasks. Principals 
viewed the portfolio process as worthwhile but somewhat burdensome. 

Teachers reported that they needed more training and clearer 
expectations about how to construct portfolios. They also indicated that 
they spent an average of 6 hours per week working on portfolios; half of 
the time was spent preparing portfolios and the other half was devoted 
to classroom portfolio activities. While time was a problem, teachers 
considered the effects on instruction to be great. Finally, teachers re- 
ported that portfolios had both positive and negative effects on the 
performance of students of low ability. Approximately 16% of the teach- 
ers reported that these students were often more successful as a result of 
portfolios. However, the majority of teachers indicated that the students 
had difficulties with the tasks. 

Reliability of the portfolio scoring was also examined for the school 
years 1991-1992 (Koretz, Stecher, & Deibert, 1992) and 1992-1.993 
(Koretz, Klein, McCaffrey, & Stecher, 1993). Spearman correlation coef- 
ficients were computed between scores of two raters. Scores for individ- 
ual pieces, as well as dimension-level (e.g., problem-solving) scores, 
were computed. For 1992-1993, the correlations for total scores for 
mathematics portfolios were .72 for grade 4 and .79 for grade 8. Simi- 
larly, correlations for writing total scores were .56 for grade 4 and .63 
for grade 8. 

Based on information from this comprehensive and ongoing evalu- 
ation, the SEA is moving to ensure greater reliability in scoring and to 
continue to provide intensive staff development to teachers. 

Littleton, Colorado 

Demographic Background 

Littleton is a suburb of Denver. There are 16,500 students in the school 
system, of whom 1,500 are identified as needing special education. The 
^hool population is primarily white and middle class. The dropout rate 
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is around 4% for the system, and 88% of Littleton's graduates go on to 
oUegp. o 

ik 

Description of Accountability System 

Littlppn's outcomes-based accountability system evolved from a strate- 
gjoptanning process that began during the 1980s. Seven broad educa- 
tional outcomes were identified (see Table 1). These outcomes are 
assessed using both norm-referenced tests and performance assess- 
ments. 

Individual school report cards are required, but schools have some 
latitude in what they report. All school report cards contain results of the 
norm-referenced tests. Performance assessment results are reported dif- 
ferently. For example, one school report card may contain a narrative 
description of its program. Diplomas are granted on the basis of success 
in attainment of outcomes. 



Assessments 

Individual schools or school clusters are allowed to identify their own 
outcomes as well as their own performance assessments, which must 
link to the seven system outcomes. As an example, Littleton High School, 
one of three district high schools, no longer uses Carnegie Units or 
required coursewoi k as part of the graduation requirements. The school 
staff has established 19 outcomes and constructed 39 performance tasks, 
as well as a set of "exit'' performance assessments, that are required to 
obtain the diploma. Throughout their 4 years of high school, students 
maintain a portfolio that contains the tasks as well as other work. The 
final task is a portfolio review conducted by a panel of school faculty 
during which the student presents and discusses his or her portfolio. 

The other high schools are developing similar programs, resulting 
in three different sets of graduation requirements in the district. Middle 
and elementary schools are also developing outcomes and performance 
assessments, either individually or as part of a cluster of schools. 

District-wide performance assessments include samples of stu- 
dent-selected nongraded or corrected work from 5th, 8th, and 11th grade 
classrooms. There are no formal rewards or sanctions for schools at- 
tached to the student performance data. The district has, however, 
supported principals for their involvement in the program through 
providing professional opportunities such as speaking at conferences. 



Exemptions 

Students identified as having disabilities may or may not take part in the 
district's total assessment program. If a student spends 50% or more of 
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participate in the assessment program. In addition, school sta" must 
determine that a group test (e. g., the Iowa Tebt of Basic Skills) w^ald be 
an approbate measure of a student's projre AUbut an estimated 1% 
to 2^@£ special education students participate in the assessments. For 
sfuerents who are exempted, the IEP b the accountability document. 



The IEP team decides which school outcomes will be 
appropriate for the student a/iC' which of the assessments 
the student will take. However, no student may receive a 
diploma without successfully completing all of the exit 
performance assessments. 



The district's IEPs are developed using the seven district outcomes. The 
IEP team decides which school outcomes will be appropriate for the 
student and which of the assessments the student will take. However, 
no student may receive a diploma without successfully completing all 
of the exit performance assessments. 

Accommod&cicns 

Some assessment accommodations, such as braille assessments and 
allowing extra time, are permitted. In addition, some IEP teams have 
looked at whether the standards should be modified; however, assess- 
ment staff aie critical of modifying the standards or providing testing 
accommodations. 



There has reportedly been a great deal of discussion 
among special and general educators regarding students 
v/ith disabilities and a recognition that most of these 
students can share the same outcomes as non-disabled 
peers. 



General Comments 

Kvrently there was a political backlash in the district, primarily centered 
i' round awarding the diploma on the basis of assessments that are 
considered to have insufficient validity and reliability. In addition, some 
parents were concerned about linking the ''process" outcomes to the 
high school diploma. In the 1993 school board election, almost all pro- 
ponents of the program were defeated. The new school board was 
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e fall of 1993, and the district has decided to move away from 
-Wased education. The superintendent has retired, and a more 
traditional approach to granting diplomas is being reinstituted. How- 
ever^^i vidual schools are permitted and encouraged to continue to use 
jp^ormance assessments for instructional improvement 

Special education teachers were not initially interested in getting 
involved in the process of developing assessments, The teachers viewed 
the activity as an additional paperwork burden. Furthermore, special 
education teachers had not been involved in identifying outcomes or 
developing initial assessments so they believed there would be little 
relevance for students with disabilities. However, as those teachers have 
become involved, there has reportedly been a great deal of discussion 
among special and general educators regarding students with disabili- 
ties and a recognition that most of these students can share the same 
outcomes as non-disabled peers. There has also been some critical scru- 
tiny of the teaching and learning process in special education, and those 
teachers report that a lot of extraneous instruction that was provided in 
special education classrooms has been eliminated. Teacher involvement 
and collaboration were cited as among the most positive aspects of the 
entire program. 



Arlington Heights, Illinois 

Demographic Background 

Arlington Heights District 214 is a grade 9 through 12 district with six 
high schools. Arlington Heights is a suburb of Chicago with 11,000 
students, about 1,000 of whom are identified as having disabilities. The 
student population is economically and culturally diverse. In one high 
school over 50 languages are spoken; another school's majority popula- 
tion is Hispanic; and one high school is over one third Jewish. 

Description of Accountability System 

The district embarked on establishing an outcomes-based education 
system in 1990. There was wide stakeholder input — including specific 
attention to community clergy and other key groups — concerning dis- 
trict plans. There are two parts to the district's assessment plan. The first 
involves 11 general learner outcomes (GLOs) (see Table 1), An interdis- 
ciplinary team of teachers and administrators developed a set of indica- 
tors for each of the GLOs. Students must demonstrate the following three 
levels of achievement for each outcome; 



26 



42 




KD^Ifal^tel knowledge base related to the outcomes. 



Level 2: Practical knowledge application with various contexts. 



In addition, students are required to demonstrate competency in 
each outcome in a minimum of three different content areas. Beginning 
in 1995, students will be awarded high school diplomas based on their 
attainment of the GLOs. The second component of the system includes 
state-mandated assessments in six program areas. 

Assessments 

Students demonstrate competency through performance assessments 
that are created and evaluated by individual teachers. There a^e no 
uniform tasks and no permanent records of student performance, other 
than those that a teacher may choose to keep. Student progress toward 
outcomes is monitored on a matrix of the GLOs and the levels of 
knowledge. Teachers may use portfolios, performances, and teacher- 
made tests. There is also a bank of teacher-made tests at the district level 
from which teachers may choose. Principals and department heads 
informally monitor classroom teachers to ensure that outcomes are being 
assessed, but there are no specific standards. The district is developing 
a generic rubric that might be applied to the individual performance 
assessments. 

A second part of the district's assessment is required by the State 
of Illinois. Each student must be assessed in six program areas (English, 
math, social science, physical education, health, fine arts). For these 
outcomes, District 214 is developing performance tasks and criterion-ref- 
erenced assessments that will be used consistently across the district. 

Exemptions 

The district has resource room programs for students with mild disabili- 
ties, one self-contained classroom for students with behavior disorders, 
one program for deaf students, and an alternative high school. Students 
with severe disabilities are educated in a regional program and are not 
included in the assessments. The SEA mandates that all students be 
assessed in the six goal areas, so the district is defining goals within each 
area that are appropriate for diverse learners. For example, English now 
includes speaking and communication and listening and looking at 
books in addition to reading and writing. Students with significant 
cognitive disabilities need only show progress toward the state goals, 
not mastery. 
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She district's GLOs, all students must demonstrate attainment; 
however, the assessments are tailored to individual students. In addi- 
tion^e interpretation of an outcome or indicator is also individualized 
bj^fe^on input from the special and general education teachers. 

Accommodations 

Any necessary accommodations are included in the assessments de- 
signed for a particular student. Since there are currently no uniform 
performance tasks, specific accommodations are not an issue. 



The district's approach is designed to be nonthreatening 
and to focus on providing students with many 
opportunities to demonstrate outcomes, thereby enriching 
the curriculum across content areas. 



General Comments 

The district has opted for a flexible assessment system in large part to 
obtain critical stakeholder endorsement. Standards, administrative bur- 
dens, record keeping and other paperwork, and school accountability 
are still minimal, and school staff are being encouraged to experiment 
and innovate. The district's approach is designed to be nonthreatening 
and to focus on providing students with many opportunities to demon- 
strate outcomes, thereby enriching the curriculum across content areas. 
Teachers have opportunities for professional development, including 
weekly meetings to share assessments and discuss issues in administer- 
ing performance assessments. 
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5. Challenges and Implications: 
Using Performance Assessments 
for Accountability 



General Issues 

The use of performance assessments as part of accountability systems 
poses a number of challenges. In general the issues of program costs and 
reliability in scoring need to be considered when using performance 
tasks as part of large-scale assessment programs. According to those 
interviewed, the areas that appeared most difficult over all were setting 
performance standards and establishing scoring rubrics that are fair, yet 
flexible. These issues are potentially more important when the results of 
the assessments are used to make decisions about high school diplomas 
or to sanction schools. An additional issue was deciding what to assess. 
Specifying the outcomes and the indicators was a difficult endeavor, 
both politically and conceptually. 

In all sites, the total assessment program included the use of more 
than performance assessment tasks. In some instances, the use of norm- 
referenced assessments was mandated by state policy. In other sites, 
traditional norm-referenced assessments were used because some edu- 
cators and policymakers believed they provided a greater sense of 
security in gauging student progress. These assessment results also 
permitted cross-district or cross-site comparisons. However, perform- 
ance assessments were considered the core of the assessment programs. 

In Maryland, the MSPAP performance tasks were administered 
consistently across the state to a sample of students. In other jurisdic- 
tions, performance assessments included some consistently adminis- 
tered tasks along with other samples of student work (e.g., portfolio 
entries). These could vary from school to school. In all sites, there was a 
major emphasis on making certain that the assessments provided many 
opportunities for students to demonstrate proficiency in an outcome 
area and that results could influence instruction either individually or at 
the school level. 
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educational outcomes considered important in their states or districts go 
weU^yond knowledge of traditional subject matter and include the 
^^lication and integration of subject matter content as well as problem 
solving, making judgments, and communicating. These educational 
outcomes could not be measured by more traditional assessments; they 
required the use of performance tasks. In addition, all those interviewed 
spoke of how performance assessments have benefited student learning 
because the assessments directly link to curriculum and can change 
classroom instruction. That feature, more than any other, v/as prompting 
policymakers to maintain performance assessments as the core of the 
assessment program. 

Teachers, Ln particular, were cited as benefiting from the use of 
performance assessments in several ways. First, in every site, classroom 
teachers are being involved in some level of designing or implementing 
the assessment program. In some districts, teachers are helping to define 
the outcomes and develop and score assessments. Teachers are working 
collaboratively to develop curriculum-embedded performance assess- 
ments. This process is promoting teacher collaboration and fostering 
self-directed professional development. Special educators have become 
involved in this collaboration and have been challenged to reevaluate 
their expectations for students with disabilities, as well as the content of 
their instruction. 



A difficult conceptual switch for some special educators is 
the move from believing that each student with a disability 
should have individualized outcomes to accepting the 
notion of a common set of outcomes across students. 



Students with Disabilities 

All of the districts and states studied (with the exception of Kentucky) 
permit students with disabilities to be exempted from participation in 
the assessment programs. Typically, decisions to exempt these students 
are made by the individual 1EP teams. Exemptions most often are made 
if the team perceives that the assessments would be particularly frustrat- 
ing or cannot be performed by the student or if the outcomes being 
assessed are not relevant to the student's educational program. Despite 
the strong impetus to include all students with disabilities in the assess- 
ments, there is still some ambiguity regarding how much individualiza- 
tion should be provided within the assessment program. A difficult 
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llawiiifclWii^h for some special educators is the move from believing 
that each student with a disability should have individualized outcomes 
to accepting the notion of a common set of outcomes across students. In 
some LpsfeKces, this means accepting the notion that students with 
jp^iSies should be permitted to have opportunities to attain the same 
valued educational outcomes expected of nondisabled students and to 
experience a broad and balanced curriculum. In the case of students with 
significant cognitive disabilities, separate or additional outcomes may 
need to be established that can set the broad program goals for all of 
these students. 



There Is a risk that many students experiencing learning 
problems, including those identified as having disabilities, 
may not receive high school diplomas if they must meet 
certain set performance standards on the assessments. 



Defining scoring standards for students with disabilities is also 
problematic in most sites inciuded in the study. In Maryland, Vermont, 
Kentucky, and Littleton, Colorado, which have standard scoring systems, 
assessment accommodations are permitted for students with disabilities, 
but these students are held to the same performance criteria as nondis- 
abled students. Individual standard setting occurs in Arlington Heights, 
Illinois. In some systems, accommodations include administering alter- 
native assessments and even defining alternative outcomes. These types 
of accommodations are made to ensure participation of students with 
significant disabilities in the assessment program and presumably to 
inform instruction. The results of these assessments are not generally used 
for program accountability or to grant diplomas. Linking diplomas to 
student assessment results has already been challenged for all students 
in Littleton, Colorado, and it is too early to know what the effects will be 
in Arlington Heights, Illinois. Nonetheless, there is a risk that many 
students experiencing learning problems, including those identified as 
having disabilities, may not receive high school diplomas if they must 
meet certain set performance standards on the assessments. 

Setting performance standards for students w th disabilities is an 
unresolved issue, and it frequently results in the exclusion of students 
with disabilities from high-stakes assessments (NCEO, 1991). While 
DeStefano (1993) has cited a number of reasons why students with 
disabilities should be included in assessment and accountability initia- 
tives, she has acknowledged the challenges. These include identifying 
meaningful outcomes, defining performance standards in sufficiently 
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s, and creating enough flexibility in the assessment system to 
arcommodate the unique needs of students with disabilities. Others (e.g., 
Jakweth & Frey, 1992; Richards, 1988; Weber & Zin, 1992) also have cited 
the^ed to have a flexible assessment system that employs multiple 
^la-gathering strategies including on-demand assessments, examples 
of student work, and teacher judgments. 

It appears from the experiences of the five states and local districts 
surveyed that the use of performance assessments as part of a larger 
accountability effort has permitted more students with disabilities to be 
included in the assessment program, Equally important, inclusion in the 
assessment program means increased attention to outcomes and in- 
creased scrutiny of the instruction provided to students with disabilities. 
Furthermore, the assessments are leading special educators to reconsider 
some of their expectations for students with disabilities and to reflect on 
the instruction they are providing. The process also provides increased 
opportunities for collaboration among general and special educators 
both in developing tasks and designing instructional and assessment 
accommodations. For all of these reasons, it appears that the use of 
performance assessments as part of large-scale programs for account- 
ability offers important opportunities for enriching the education of 
students with disabilities. 



Inclusion in the assessment program means increased 
attention to outcomes and increased scrutiny of the 
instruction provided to students with disabilities. 



This is of critical importance if we are to view performance assess- 
ments as more than a fad approach to evaluating student performance. 
Rather, performance assessments appear to provide educators and poli- 
cymakers the opportunity to define and assess educational outcomes 
that are more authentic and reflect higher-order thinking skills. At the 
same time, the assessments can link to classroom instruction and pro- 
gram improvement. Teacher and student involvement in the assessment 
process is increased, and there can be a clearer understanding of what is 
expected in the teaching/ learning process. 

Despite the positive opportunities offered by performance assess- 
ments, their use as part of large-scale accountability systems is troubled 
by issues related to scoruig reliability and the resources required for 
administration, scoring, and professional development. Nonetheless, if 
the experiences of the five sites are any indications, there is a commit- 
ment and willingness to overcome the obstacles and maintain perform- 
ance assessments as a featured part of an assessment program. 
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E1IC DoeiKit feprotek §gr?k 

^ *€V The ERIC/OSEP Special Project 



The ERIC/OSEP Special Project at The Council for Exceptional Children 
facilitates communication among researchers sponsored by the Office of 
Special Education Programs (OSEP) in the U.S. Department of Educa- 
tion, and it disseminates information about special education research 
to audiences involved in the development and delivery of special edu- 
cation services. These audiences include 

• Teachers and related services professionals. 

• Teacher trainers. 

• Administrators. 

• Policymakers. 

• Researchers. 

The activities of the ERIC/OSEP Special Project include tracking 
current research, planning and coordinating research conferences, and 
developing a variety of publications that synthesize or summarize recent 
research on critical issues and topics. Each year, the Special Project hosts 
a conference attended by research project directors sponsored by OSEP. 
Throughout the year, it holds research forums and workgroups to bring 
together experts on emerging topics of interest. Focus groups repre- 
senting the Special Project's audiences are held to inform both OSEP and 
the Special Project of audience information needs and to enhance the 
utility of publications produced by the Special Project. These publica- 
tions include an annual directory of research projects as well as publica- 
tions about current research efforts. 

The ERIC/OSEP Special Project is funded under a three-party 
contract between The Council for Exceptional Children, the Office of 
Special Education Programs, and the Office of Educational Research and 
Improvement, U. S. Department of Education. Under this contract, OSEP 
funds the ERIC/OSEP Special Project, and OERI funds the ERIC Clear- 
inghouse on Disabilities and Gifted Education. The ERIC Clearinghouse 
on Disabilities and Gifted Education is one of 16 clearinghouses of the 
Educational Resources Information Center (ERIC) system, which main- 
tains a database of over 440,000 journal annotations and 340,000 docu- 
ment abstracts concerning education. The ERIC Clearinghouse on 
Disabilities and Gifted Education gathers and disseminates information 
on all disabilities and on giftedness across age levels. 
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